A Framework for Capture and Synthesis of High Resolution Facial Geometry and Performance

We present a framework that captures and synthesizes high resolution facial geometry and performance. In order to capture highly detailed surface structures, a theory of fast normal recovery using spherical gradient illumination patterns is presented to estimate surface normal maps of an object from either its diffuse or specular reflectance, simultaneously from any viewpoints. We show that the normal map from specular reflectance yields the best record of detailed surface shape, which can be used for geometry enhancement. Moreover, the normal map from the diffuse reflectance is able to produce a good approximation of subsurface scattering. Based on the theory, two systems are developed to capture high resolution facial geometry of a static face or dynamic facial performance. The static face scanning system consists of a spherical illumination device, two singlelens reflex (SLR) cameras and a video projector. The spherical illumination device is used to cast spherical gradient patterns onto the subject. The captured spherical gradient images are then turned into surface normals of the subject. The two cameras and one projector are used to build a structured-light-assisted two-view stereo system, which acquires a moderate resolution geometry of the subject. We then use the acquired specular normal map to enhance the initial geometry based on an optimization process. To further analyze how facial geometry deforms during performance, we build another facial performance capture system, which is analogous to the previous face scanning system, but employs two high-speed video cameras and a high-speed projector. The system is able to capture 30 facial geometry measurements per second. A novel method named polynomial displacement maps is presented to cooperate motion capture with real-time face scans, so that realistic facial deformation can be modeled and synthesized. Finally, we present a real-time relighting algorithm based on spherical wavelets for rendering realistic faces under modern GPU architecture.